Papers with prohibitive computational cost
Once is Enough: A Light-Weight Cross-Attention for Fast Sentence Pair Modeling (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies suggest that transformer-based models perform cross-attention over input pairs, leading to computational cost. |
| Approach: | They propose a lightweight cross-attention mechanism that performs query encoding only once while modeling the query-candidate interaction in parallel. |
| Outcome: | The proposed model speeds up sentence pairing by over 113x while achieving comparable performance as the more expensive models. |
Training for Gibbs Sampling on Conditional Random Fields with Neural Scoring Factors (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in NLP focus on simple approaches to model the output label space . graphical models are often limited to (heuristic) greedy search and its variants . |
| Approach: | They propose an approach for efficiently training and decoding hybrids of graphical and graphical models based on Gibbs sampling. |
| Outcome: | The proposed approach improves on Dutch and Dutch with graphical models . the proposed model improves over a strong baseline on three languages . |
Interleaved Latent Visual Reasoning with Selective Perceptual Modeling (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to interleaved reasoning are limited by the cost of re-encoding pixel-dense images. |
| Approach: | They propose a framework that unifies dynamic state evolution with precise perceptual modeling. |
| Outcome: | The proposed framework outperforms existing approaches on multimodal reasoning benchmarks. |
Data Efficient RLVR via Off-Policy Influence Guidance (2026.acl-long)
Copied to clipboard
Erle Zhu, Dazhi Jiang, Yuan Wang, Xujun Li, Jiale Cheng, Yuxian Gu, Yilin Niu, Aohan Zeng, Jie Tang, Minlie Huang, Hongning Wang
| Challenge: | Existing data selection methods for RLVR are heuristic-based, lacking theoretical guarantees and generalizability. |
| Approach: | They propose an off-policy influence estimation method that approximates data influence using offline trajectories. |
| Outcome: | The proposed method reduces the computational cost of policy rollouts and improves storage and computation efficiency. |
SELECting over Tokens: Curating Pre-training Data at Scale via Token Classification (2026.acl-long)
Copied to clipboard
Xin Tong, Weidong Zhang, Jiaang Li, Haibin Chen, Shilei Liu, Langming Liu, Kangtao Lv, Yujin Yuan, Wenbo Su, Bo Zheng
| Challenge: | Existing pipelines rely on expert-crafted heuristic rules, which lack content-aware, fine-grained noise detection. |
| Approach: | They propose a framework that reframes data refinement as a highly efficient token classification task. |
| Outcome: | The proposed framework outperforms existing pipelines on benchmarks and is 2.5x faster at inference. |